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Abstract 

The present study focused on the analysis of listening sections of two international English proficiency tests, i.e. 
IELTS and TOEFL tests, and one local English proficiency test, i.e. TOLIMO from pragmatic perspective. An 
attempt was made to explore the areas of pragmatic knowledge presented, and to assess test takers’ pragmatic 
knowledge. For this purpose, 250 items from each of these three proficiency tests were collected and analyzed. 
IELTS and TOLIMO tests were taken from preparation textbooks available in the market, and TOEFL tests were 
taken from the tests administered from 2000 to 2004. To elucidate what areas of pragmatic knowledge was 
involved in each item, Jung’s (2002) classification of components of pragmatic knowledge was used. In this 
study, pragmatic knowledge, includedthe ability to perform speech acts, the ability to convey and interpret 
non-literal meanings, the ability to perform politeness functions, the ability to perform discourse functions, and 
the ability to use cultural knowledge. The results of the study showed that TOLIMO, TOEFL and IELTS tests are 
able to assess test takers’ pragmatic knowledge; however, higher instances of pragmatic knowledge components 
were involved in TOEFL and TOLIMO test. 

Keywords: listening comprehension, communicative competence, pragmatic knowledge, IELTS, TOEFL 

1. Introduction 

Reading comprehension is the application of a skill that evolved for other purposes (listening or oral 
comprehension) to a new form of input (text). Whereas oral comprehension seems to develop “naturally” with 
minimal intentional involvement, reading comprehension is more challenging and requires deliberate instruction. 
Human beings have been using oral form of languages for 100,000 years or more (Donald, 1991), and almost all 
humans do it; reading comprehension has only been practiced for 5,000 years, and for most of that time. Most 
human beings did not do it (Olson, 1994). Thus, one of the sources of the difficulty of reading comprehension is 
its novelty (Kirby, 2006). 

Reading comprehension is the process of constructing meaning from the text. The goal of all reading instructions 
is ultimately targeted at helping a reader to comprehend a given text. Reading comprehension involves at least 
two people; the reader and the writer. The process of comprehending involves decoding the writer’s words and 
then using background knowledge to construct an approximate understanding of the writer’s message (Kirby, 

2006, p. 161). 

One of the components of language proficiency knowledge is pragmatic knowledge. To be proficient in a 
language, EFL learners need to be pragmatically competent. Pragmatic knowledge, previously a neglected area 
in the realm of SLA, has increasingly taken more and more attention in recent years (Bachman, 1990; Garcia, 
2004). That is, because being considered as a proficient second language, a user means not only having 
grammatical, lexical and phonological aspects but also having pragmatic knowledge (Corsetti, 2010). According 
to Van Dijk (1977), the pragmatic comprehension is different from linguistic comprehension because it calls for 
contextual informationsuch as the role played by interlocutors and status of them, the physical setting of the 
conversation, and the types of communicative acts that may occur in that context. Crystal (2008) defines 
pragmatics as “the study of language from the point of view of the users, especially of the choices they make, the 
constraints they encounter in using language in social interaction, and the effects their use of language has on the 
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other participants in an act of communication” (p. 379). Pragmatics is concerned with rules and principles of 
language use in social contexts (Tan, 1994). Also, pragmatics involves using the language to show the linguistic 
roles increased by the social functions of language (Brown & Levinson, 1987). According to Flowerdew and 
Miller (2005), in order to comprehend a spoken message, four main types of knowledge are needed: phonology; 
syntax; semantics; and pragmatics. Pragmatics means to understand the meaning of produced utterances in 
particular situations. 

In this study, the reading section of Test of English as a Foreign Language (TOEFL), International English 
Language Testing System (IELTS), and a local English proficiency, TOLIMO (The Test of Language by the 
Iranian Measurement Organizations), were taken into accountregarding the involvement of components of 
pragmatic knowledge. 

Uso-Juan and Martinez-Flor (2006) have elaborated the role of reading in communicative competence 
framework. They believe that linguistic, strategies, intercultural, pragmatic competence, as well as discourse 
competence are related to reading ability. Another type of competence, which is the most pertinent component, is 
discourse competence which involves “the knowledge of written discourse features such as markers, cohesion 
and coherence as well as formal schemata with reference to the particular communicative goal and context of the 
written text” (p. 268). Linguistic competence is reported to be lowest level of ability required to understand a 
piece of text. Vocabulary is a subsection of this kind of competence, which is of high importance in the reading 
ability. Strategic competence deals with both learning and communication strategies. The former deals with 
metacognitice, cognitive and socioaffective measures which are taken by the reader to have a better 
understanding of the text, and the latter has to do with using reading strategies to make up for interpretation 
deficiencies. Intercultural competence refers “to the knowledge of how to interpret written texts appropriately 
within their sociocultural context” (270). The last and the most relevant competence to the present study is the 
pragmatic competence, which involves the understanding of the meaning which is beyond the locutionary; in 
other words, it deals with the illocutionary force of utterances. The knowledge of context is of paramount 
significance here to attach the right illocutionary force to an utterance. 

As it was already mentioned, one of the language proficiency components, which is influential in the process of 
comprehending a text is the pragmatic knowledge. Having syntactic and semantic knowledge does not guarantee 
students’ second language communicative competence. Students also need to possess pragmatic knowledge in 
order to be considered as a proficient speaker of a second language. “Inability to match utterances with contexts 
in which they are appropriate has affected students’ overall communicative competence” (Fakeye David, 2008, 
p.73). Not having enough pragmatic knowledge leads to misunderstanding and misinterpretation, which may 
break down the whole communication process? TOEFL, IELTS, and TOLIMO are said to be able to assess 
testees’ pragmatic competence. But, to what extent and what areas of pragmatic knowledge are assessed in these 
three proficiency tests? Therefore, themain objective of this study is to investigate to what extent pragmatic 
knowledge is assessed in reading sections of TOEFL, IELTS, and TOLIMO. 

The present study tries to fill the gap in the literature by probing into the pragmatic knowledge assessed in three 
proficiency tests, namely TOEFL, IELTS, and TOLIMO. No study has ever investigated the ability of reading 
sections of the international proficiency tests in assessing test takers’ pragmatic knowledge. 

1.1 Research Questions 

The present study tries to answer the following questions: 

1) To what extent can reading section of TOEFL assess pragmatic knowledge? 

2) What areas of pragmatic knowledge are assessed in reading section of TOEFL? 

3) To what extent can reading section of IELTS assess pragmatic knowledge? 

4) What areas of pragmatic knowledge are assessed in reading section of IELTS? 

5) To what extent can reading section of TOLIMO assess pragmatic knowledge? 

6) What areas of pragmatic knowledge are assessed in reading section of TOLIMO? 

7) Which test assesses pragmatic knowledge of test takers more? IELTS, TOEFL, or TOLIMO? 

2. Literature Review 

According to Jung (2002), a person to be considered as a pragmatically competent language user needs to have 
the following aspects: 

1) The Ability to Perform Speech Acts: speech act was developed by Austin (1962) and later more elaborated by 
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Searle (1969). At the beginning of “How to Do Things with Words”, J. L. Austin asserts that “the business of a 
[sentence] can only be to ‘describe’ some state of affairs, or to ‘state some fact’, which it must do either truly or 
falsely” (1962, p. 6). 

A few years later in 1969, Searle worked more on speech acts and give functions to them. He then classified 
them into five categories, which according to Yule (2000) are as follows: 

a) Declaratives, which are those kinds of speech acts that change the world through the utterances. For example: 

• 1 now pronounce you man and wife. 

b) Representatives, which are those kinds of speech acts that state what the speaker believes to be the case or not. 
Statements of fact, assertions, and conclusions belong to this category. For example: 

• William Faulkner wrote The Sanctuary. 

c) Expressives, which are those kinds of speech acts that state what the speaker feels. They express various 
psychological states such as likes, dislikes, joy, sorrow, etc. For example: 

• Wow, how beautiful you are! 

• I really love your new dress. 

• The meal was delicious. 

d) Directives, which are those kinds of speech acts through which the speaker gets someone to do something. 
Commands, orders, requests, and suggestions belong to this category. For example: 

• Don’t hit your sister. 

e) Commissives, which are those kinds of speech acts that speakers use to commit themselves to some future 
action such as promises, threats, refusals, etc. For example: 

• I’ll help you. 

2) The Ability to Convey and Interpret Non-literal Meanings : This ability is directly related to the Grice’s 
cooperative principles and the meaning of implicature. According to Stanford Encyclopedia of Philosophy by 
Wayne Davis (forthcoming), “Implicature denotes either (i) the act of meaning, implying, or suggesting one 
thing by saying something else, or (ii) the object of that act.” Conversational implicatures have become one of 
the main issues in the study of pragmatics. 

Grice (1975, p. 45) maintains that the dominant principle in conversation is cooperative principle: “Make your 
conversational contribution such as is required, at the stage at which it occurs, by the accepted purpose or 
direction of the talk exchange in which you are engaged.” Grice, who coined the term “implicature,” and 
classified the phenomenon, developed an important theory to explain and predict conversational implicatures, 
and describe how they used and are understood. 

3) The Ability to Perform Politeness strategies'. According to Holmes (2008), being polite is a complicated 
business in any language. It is difficult to leam because it involves understanding not just the language, but also 
the social and cultural values of the community. She also mentions that “generally speaking politeness involves 
taking into account the feeling of others.” (p. 281). According to major studies of politeness (Brown & Levinson, 
1987; Lakoff, 1973; Leech, 1983), linguistic expressions display different degrees of politeness. 

Politeness, sometimes, is attached to indirectness (Austin, 1962; Brown & Levinson, 1987; Srinarawat, 2005). 
And, Indirectness is prevalent in all human languages (Srinarawat, 2005). Brown and Levinson (1987) classify 
direct speech acts as Face Threatening Acts (FTAs). In the direct speech act, the speaker employs a sentence 
whose meaning explicitly provides the hearer with the content of the intended act. Indirectness, on the other hand, 
refers to the speech act in which the expressed meaning of an utterance does not match the speaker’s implied or 
intended meaning (Srinarawat, 2005). An indirect speech act necessitates the speaker’s and the hearer’s common 
background knowledge and the ability to make inferences on the listener’s part (Rabinowitz, 1993). 

According to Watts (2003), investigation of directness or indirectness in speech act realizations is pertinent to 
politeness. Leech (1983) argues that indirectness usually comes hand in hand with politeness and that the “more 
indirect an illocution is, the more diminished and tentative its force tends to be” (p. 108). In probing into 
directness/indirectness and its effect on politeness, Blum-Kulka (1987) concludes that direct strategies are often 
taken as impolite since they lack a concern for face. On the contrary. Brown and Levinson (1987) maintain that 
the level of directness is to be ruled by universal principles of politeness. 

4) The ability to carry’ out discourse functions. Since the last few decades, linguists have recognized that 
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communication is not based solely on sentence-level (lexis and sentence structure) and that the investigation of 
language and language learning should incorporate longer stretches of text or what is known as discourse. Many 
linguists have investigated the relationship between sentences in a text and labeled this relationship as texture. A 
set of sentences constitute a text if there is a relationship within and between the sentences; otherwise, they 
would be only a group of unrelated sentences. “The connections between sentences are called cohesive relations” 
(Yule & Brown, 1989, p. 191). Different parts of a text (or a conversation or any stretch of language) are 
interrelated in different forms. Sometimes the underlying semantic relations between the sentences and 
propositions have the cohesive power and indicate texture (coherence); however, in many cases there are some 
linguistic elements which manifest the relationship between the facts and propositions in a text (cohesion). The 
most frequently studied markers signaling coherence relations are discourse markers. 

Fraser (1999) has defined DMs “as a pragmatic class, lexical expressions drawn from the syntactic classes of 
conjunctions, adverbials, and prepositional phrases.” He asserts that with certain exceptions, they signal a 
relationship between the segment they introduce, S2, and the prior segment, SI. Their core meaning is not 
conceptual but procedural, and their more specific interpretation is ‘negotiated’ by the context, both linguistic 
and conceptual. 

Fraser (1999) has divided DMs in four important groups: Contrastive Discourse Markers (CDM), Elaborative 
Discourse Markers (EDM), Inferential Discourse Markers (IDM), and Temporal Discourse Markers (TDM). 

5) Cultural knowledge: Culture is what people “must know in order to act as they do, make the things they make, 
and interpret their experience in the distinctive way they do” (Quinn & Holland, 1987, p. 4). According to Bloch 
(1991), culture, which is studied in the realm of anthropology, is defined as what people must be acquainted with 
in order to function reasonably and effectively in their social environment. Social environment consists of social 
organizations and behaviors that are the instruments through which people relate to each other. “A society’s 
culture consists of whatever it is one has to know or believes in order to operate in a manner acceptable to its 
members, and to do so in any role that they accept for anyone one of themselves” (Goodenough, 1957, p. 167). 
So, cultural knowledge is “socially acquired: the necessary behaviors are learned and do not come from any kind 
of genetic endowment” (Wardhaugh, 2008, p. 216). 

When exploring culture, we have to consider concepts such as schema (plural, schemata), frame, and script. Yule 
(2000) argues that “a schema is a pre-existing knowledge structure in memory, (p. 85)” which can be either static 
or dynamic. When schema is fixed and static, it is called a frame. A frame is shared within a social group, which 
is like a prototypical version. For instance, in a frame for a restaurant, tables, plates, fork, spoon, and food will 
be come into mind. According to Yule (2000), a more vibrant type of schemata is called scrip, which is a 
pre-existing knowledge arrangement involving event sequences. Scripts are used to construct interpretations of 
accounts of what happened, and is a way of recognizing some expected sequence of actions in an event. 

Since this study is the first in the pragmatic area of evaluating reading sections in TOEFL, IELTS, and TOLIMO 
tests, the researcher did not find any related empirical studies neither in TOEFL, IELTS, and TOLIMO, nor in 
any other type of universal language proficiency tests. 

3. Methodology 

In order to collect data for the purpose of this study, qualitative method was used. Regarding the design, 
descriptive design was taken into account for data collection. 

3.1 Material 

Materials used in this study included reading sections of two internationally-accepted English proficiency tests, 
IELTS and TOEFL, and a local proficiency test, TOLIMO. 

IELTS Reading Section: Because the original IELTS tests were not accessible, the researcher used and analyzed 
IELTS preparation tests available in the market. Five series of reading comprehension tests were adopted from 
Practice Tests for IELTS 1 (Jakeman & McDowell, 1997), Practice Tests for IELTS 3 (2004), and insight into 
IELTS (Jakeman & McDowell, 2002). In general, 250 items were taken and analyzed in this study. 

TOEFL Reading Section: The Reading Comprehension section included 50 questions onreading passages. To 
determine test takers’ ability to understand written English, they must answer multiple questions about the ideas 
and the meanings of words in reading passages. In this study, five series of TOEFL from 2000 to 2004 were 
investigated. As a whole, 250 questions were used and analyzed. 

TOLIMO Reading Section: This test is similar to TOEFL. The Reading Comprehension section included 50 
questions about reading passages. There are 5-6 passages and 8-12 questions about each passage. To determine test 
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takers’ ability to understand written English, they should answer multiple-choice questions about the ideas and the 
meanings of words in reading passages. In this study, 5 series of preparation tests for TOLIMO were investigated. 
As a whole, 250 questions were under full scrutiny. 

3.2 Procedure 

In this study, the reading sections of IELTS, TOEFL, and TOLIMO tests were analyzed according to the five 
areas of pragmatic knowledge mentioned in Jung (2002). Each item of these tests was analyzed one by one. In 
this section, the five areas of pragmatic knowledge are explained and exemplified. One of the main features of a 
research is a high degree of consistency. To estimate the inter-coder reliability, Spearman Correlation test was 
run. These tests were analyzed by three experts to make sure that the coding process is done precisely. 

3.3 Data Analysis 

To answer the seven researchquestions, a series of Chi-square was calculated. As we were dealing with 
frequency of types of levels, two types of chi-square tests were used. A chi-square test for goodness of fit was 
usedfor variables with one category and chi-square test of independence was utilized for variables with more 
than one category. 

4. Results 

Before data analysis, we should know that, for all statistical analyses in this study, .05 was used as the alpha 
level at which findings were considered to be significant. Several statistical tests were employed to address the 
different research questions. The data were analyzed using the Statistical Package for the Social Sciences (SPSS 
16). 

Analysis of Speech Acts in TOEFL: Speech acts were assessed in 49.6% of items (N=124). Commissives had the 
lowest number of speech acts (N=13). Representatives were used in 29 items and expressives in 40 items. And 
the highes test number of assessed speech acts belonged to dirrectives (N=42). Also, there was a significant 
difference in the frequency of speech act categories, % 2 =17.097, df=3, p=0.00 1. 


Table 1. Chi-square for the distribution of speech acts in TOEFL 


Speech acts 

Chi-Square 

17.097 

Df 

3 

Asymp. Sig. 

.001 


Analysis of Discourse Markers in TOEFL: Discourse markers were used in 132 items of TOEFL (52.8%). 
Elaborative discourse markers were assessed in 33 items (13.2%). Contrastive discourse markers had 64 
instances in TOEFL test (25.6%). and temporal discourse markers had the frequency of 22 (8.8%). The least 
discourse markers used was inferential (N=13, 5.2%). There was a significant difference between the category of 
discourse markers used in TOEFL (x 2 =44.909, df=3,/»<0.05). 


Table 2. Chi-square tests for the distribution of discourse markers in TOEFL 


Discourse Markers 

Chi-Square 

44.909 

Df 

3 

Asymp. Sig. 

.000 


Analysis of Politeness Strategies in TOEFL: In 7.2% of items in TOEFL, politeness functions were included 
(N=18). Direct strategies had been used in 8 items (3.2%) and indirect strategies in 10 items (4%). As a whole 
there were 18 instances of using politeness strategies in TOEFL tests. However, as can be seen in Table 3, there 
was no significant difference in inclusion of politeness strategies. (p>0.05) 
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Table 3. Chi-square tests for the distribution of politeness strategies in TOEFL 


Politeness 

Chi-Square 

.222 

df 

1 

Asymp. Sig. 

.637 


Analysis of Implicature in TOEFL: In TOEFL test, only in 113 items non-literal meaning has been assessed 
(45.2%). The maxim of quantity was assessed in 23 items (9.2%), and the maxim of manner in 31 items (12.4%). 
Maxims of quality 22 (8.8%) and relevance 37 times were used in TOEFL tests (14.8%). Yet, no significant 
difference was found in the category of implicature (x 2 =05.336, df=l, /;>().05). 


Table 4. Chi-square tests for the distribution of implicature in TOEFL 


Implicature 

Chi-Square 

5.336 

df 

1 

Asymp. Sig. 

.149 


Analysis of Speech Acts in IELTS: Speech acts were assessed in 22.4% of items (N=56). Commissives had the 
highest number of speech acts (N=26). Representatives were used in 21 items and expressives in 6 items. And, 
the lowest number of assessed speech acts belonged to dirrectives (N=3). Also, there was a significant difference 
in the frequency of speech act categories (x 2 =27.000, df=3, p=0.00). 


Table 5. Chi-square for the distribution of speech acts in IELTS 


Speech acts 

Chi-Square 

27.000 

df 

3 

Asymp. Sig. 

.000 


Analysis of Discourse Markers in IELTS: Discourse markers were used in 33 items of IELTS (13.3%). 
Elaborative discourse markers were assessed in 15 items (6%). Contrastive discourse markers and inferential 
discourse markers each had the frequency of 8 (3.2%). The least discourse markers used was temporal (N=2, 
0.8%). There was a significant difference between the category of discourse markers used in IELTS (x 2= 10.723, 
df=3,/?<0.05). 


Table 6. Chi-square tests for the distribution of discourse markers in IELTS 


Discourse Markers 

Chi-Square 

10.723 

df 

3 

Asymp. Sig. 

.016 


Analysis of Politeness Strategies in IELTS: In 11.6% of items in IELTS, politeness functions were included 
(N=29). Direct strategies had been used in 16 items (6.4%) and indirect strategies in 13 items (5.2%). As a whole 
there were 29 instances of using politeness strategies in IELTS tests. Flowever, as can be seen in Table 7, there 
was no significant difference in inclusion of politeness strategies. (p>0.05) 


213 




www.ccsenet.org/elt 


English Language Teaching 


Vol. 8, No. 5; 2015 


Table 7. Chi-square tests for the distribution of politeness strategies in IELTS 


Politeness 

Chi-Square 

.310 

df 

1 

Asymp. Sig. 

.577 


Analysis of Implicature in IELTS: In IELTS test, only in 19 items non-literal meaning has been assessed (7.6%). 
The maxim of quantity was assessed in 10 items (4%), and the maxim of manner in 9 items (3.6%). Yet, no 
significant difference was found in the category of implicature (x 2 =0.053, df= 1, £>>0.05). 


Table 8. Chi-square tests for the distribution of implicature in IELTS tests 


Implicature 

Chi-Square 

.053 

df 

1 

Asymp. Sig. 

.819 


Analysis of Speech Acts in TOLIMO: Speech acts were assessed in 45.6% of items (N=l 14). Commissives had 
the lowest number of speech acts (N=22). Representatives were used in 29 items and expressives in 30 items. 
And the highest number of assessed speech acts belonged to directives (N=33). Also, there was no significant 
difference in the frequency of speech act categories (% 2 =17.097, df=3, p=0426). 


Table 9. Chi-square for the distribution of speech acts in TOLIMO 


Speech acts 

Chi-Square 

2.788 

df 

3 

Asymp. Sig. 

.426 


Analysis of Discourse Markers in TOLIMO: Discourse markers were used in 122 items of TOLIMO (48.8%). 
Elaborative discourse markers were assessed in 59 items (23.6%). Contrastive discourse markers had 20 
instances in TOLIMO test (8%). and temporal discourse markers had the frequency of 24 (9.6%). The least 
discourse markers used was inferential (N=19, 7.6%). There was a significant difference between the category of 
discourse markers used in TOLIMO (x 2 =35.967, df=3,/><0.05). 


Table 10. Chi-square tests for the distribution of discourse markers in TOLIMO 


Discourse Markers 

Chi-Square 

35.967 

df 

3 

Asymp. Sig. 

.000 


Analysis of Politeness Strategies in TOLIMO: In 6.4% of items in TOLIMO, politeness functions were included 
(N=16). Direct strategies had been used in 9 items (3.6%) and indirect strategies in 7 items (2.8%). As a whole, 
there were 16 instances of using politeness strategies in TOLIMO tests. However, as can be seen in Table 11, 
there was no significant difference in inclusion of politeness strategies. (p>0.05) 
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Table 11. Chi-square tests for the distribution of politeness strategies in TOLIMO 


Politeness 

Chi-Square 

.250 

df 

1 

Asymp. Sig. 

.617 


Analysis of Implicature in TOLIMO: In TOLIMO test, only in 110 items non-literal meaning has been assessed 
(44%). The maxim of quantity was assessed in 23 items (9.2%), and the maxim of manner in 31 items (12.4%). 
Maxims of quality 20 (8%) and relevance 36 times were used in TOLIMO tests (14.4%). Yet, no significant 
difference was found in the category of implicature ()f=5.855, df=l,/»>0.05). 


Table 12. Chi-square tests for the distribution of implicature in TOLIMO tests 


Implicature 

Chi-Square 

5.855 

df 

1 

Asymp. Sig. 

.119 


5. Analysis of Different Categories in TOEFL, IELTS, and TOLIMO 

The following table reveals the frequency of different categories of speech acts, expressive, representatives, 
directives, and commissives, in TOEFL, IELTS, and TOLIMO. 


Table 13. Frequency of speech acts in TOEFL, IELTS, and TOLIMO 




Expressives 

Representatives 

Directives 

Commissives 

Total 


TOEFL 

40 

29 

42 

13 

124 

Test 

IELTS 

6 

21 

3 

26 

56 


TOLIMO 

30 

29 

33 

22 

114 

Total 


76 

79 

78 

61 

294 


A chi-square test for independence indicated that there was a significant difference between TOLIMO, TOEFL, 
and IELTS with regard to assessing speech acts, (x 2 =46.674, df=6,/><0.05). 


Table 14. Difference between TOLIMO, IELTS, and TOEFL with regard to speech acts 



Value 

df 

Asymp. Sig. (2-sided) 

Pearson Chi-Square 

46.674 

6 

.000 

Likelihood Ratio 

48.385 

6 

.000 

Linear-by-Linear Association 

.924 

1 

.337 

N of Valid Cases 

293 




5 .1 Discourse Markers 

Table 15 shows a sharp contrast between frequency of items in TOLIMO, TOEFL, and IELTS with regard to 
discourse markers. As can be seen in Table 15, in TOEFL out of 250 tests 124 items assessed discourse markers, 
while only 56 out of 250 items of IELTS were involved in assessing test takers’ knowledge of discourse markers. 
Also, 114 instances of discourse markers were found in 250 TOLIMO tests. 
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Table 15. Frequency of discourse markers in IELTS, TOEFL, and TOLIMO 




Discourse markers 



_Total 

CDM 

EDM 

IDM 

TDM 



TOEFL 

64 

33 

13 

22 

132 

Test 

IELTS 

8 

15 

8 

2 

33 


TOLIMO 

20 

59 

19 

24 

122 

Total 


62 

44 

19 

25 

287 


A chi-square test for independence indicated that there was a significant difference between TOLIMO, IELTS, 
and TOEFL tests with regard to assessing discourse markers (x 2 =38.238, df=6,p<0.05). 


Table 16. Difference between TOLIMO, IELTS, and TOEFL with regard to discourse markers 



Value 

df 

Asymp. Sig. (2-sided) 

Pearson Chi-Square 

38.238 

6 

.000 

Likelihood Ratio 

39.564 

6 

.000 

Linear-by-Linear Association 

10.960 

1 

.001 

N of Valid Cases 

287 




Table 17 depicts the frequency of each politeness strategies in IELTS and TOEFL. There are two kinds of 
politeness principles: direct and indirect. 


Table 17. Frequency of politeness strategies in IELTS, TOEFL, and TOLIMO 




Politeness 

Total 

Direct 

Indirect 



TOEFL 

8 

10 

18 

Test 

IELTS 

16 

13 

29 


TOLIMO 

9 

7 

16 

Total 


33 

30 

63 


A chi-square test for independence indicated no significant difference between the frequency of IELTS and 
TOEFL in terms of assessing politeness function x 2= 0.641, df=2, £>>0.05 (see Table 18). 

Table 18. Difference between TOLIMO, IELTS, and TOEFL with regard to politeness functions 



Value 

df 

Asymp. Sig. (2-sided) 

Pearson Chi-Square 

0.641 

2 

.726 

Likelihood Ratio 

0.641 

2 

0.726 

Linear-by-Linear Association 

0.003 

1 

0.954 

N of Valid Cases 

63 




5.2 Implicature 

Table 19 shows the frequency of category of non-literal meaning. 
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Table 19. Frequency of implicature in IELTS, TOEFL, and TOLIMO 



Implicature 




— Total 



Flouting Quality 

Flouting Quantity 

Flouting Relation 

Flouting Manner 



TOEFL 

22 

23 

37 

31 

123 

Test 

IELTS 

0 

10 

0 

9 

19 


TOLIMO 

20 

23 

36 

31 

110 

Total 


42 

56 

73 

71 

242 


A chi-square test for independence indicated that there was a significant difference between IELTS, TOEFL, and 
TOLIMO tests in terms of assessing non-literal meaning, x 2== 17.790, df=6,p<0.05. 

Table 20. Difference between IELTS, TOEFL, and TOLIMO with regard to non-literal meaning 



Value 

df 

Asymp. Sig. (2-sided) 

Pearson Chi-Square 

17.790 

6 

0.007 

Likelihood Ratio 

23.930 

6 

0.001 

Linear-by-Linear Association 

0.524 

1 

.469 

N of Valid Cases 

242 




5.3 Pragmatic Knowledge 

Table 21 depicts the frequency of category of pragmatic knowledge in TOEFL and IELTS. 
Table 21. Frequency of component of pragmatic knowledge in IELTS, TOEFL, and TOLIMO 




Pragmatic knowledge 



Total 

Speech acts 

Discourse markers 

Politeness functions 

Implicature 


TOEFL 

124 

132 

18 

123 

397 

Test 

IELTS 

56 

33 

29 

19 

137 


TOLIMO 

114 

122 

16 

110 

362 

Total 


204 

287 

63 

253 

896 


A chi-square test for independence indicated that there was a significant difference between the TOEFL, IELTS, 
and TOLIMO with regard to assessing pragmatic knowledge, x 2= 64.088, df=6,/><0.05. 


Table 22. Difference between IELTS, TOEFL, and TOLIMO with regard to pragmatic knowledge 



Value 

df 

Asymp. Sig. (2-sided) 

Pearson Chi-Square 

64.088 

6 

.000 

Likelihood Ratio 

53.688 

6 

.000 

Linear-by-Linear Association 

2.843 

1 

.000 

N of Valid Cases 

896 




6. Discussion 

Ql. To what extent can reading section of TOEFL assess pragmatic knowledge? 

The purpose of this research question was to find the frequency and the percentage of the involvement of 
pragmatic knowledge in reading section of TOEFL test. This research question tries to find that in how many of 
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items in ihe reading section of TOEFL test pragmatic knowledge has been assessed. In this study 250 items of 
TOEFL test were analyzed. In 68.4 percent of the test items pragmatic knowledge has been evaluated at least 
once (N=171). The results showed that pragmatic knowledge categories were used 397 times. It was found that 
in 62 items, pragmatic knowledge has been used once (24.8%). It has been shown that in TOEFL test there are 
items in which more than one areas of pragmatic knowledge has been assessed (N=109, 43.6%). A test which 
assesses pragmatic knowledge in 68.4% of its items can be considered as a pragmatic test. Therefore, it is clear 
that TOEFL test is a pragmatically-oriented English proficiency test. 

Q2. What areas ofpragmatic knowledge are assessed in reading section of TOEFL? 

In this study, attempts have been made to show the categories and subcategories of pragmatic knowledge 
involved in TOEFL test. 

Speech acts : With regard to the first category, the researcher used Searle’s (1975) taxonomy of illocutionary 
speech acts in order to consider this area of pragmatic knowledge. For TOEFL test, out of 250 items, 124 
instances of speech acts were found. So nearly in the half of the items speech acts were used. 

Discourse markers: With regard to the category of discourse functions, the researcher used Fraser’s (1999) 
classification of discourse markers. Out of the 250 items in TOEFL test, there were 132 instances of discourse 
markers. Contrastive discourse markers have been used more than other types of discourse markers (N=64). 
Inferential discourse markers with frequency of 13 was the least assessed discourse markers. In 33 items, 
elaborative discourse markers were assessed. And, temporal discourse markers were used in 22 items in TOEFL 
test. 

Politeness strategies: Politeness strategies are another component of pragmatic knowledge which were analysed 
based on directness and indirectness. Among 250 items in TOEFL test, there were 18 items which were related to 
the politeness functions. Direct strategies were used in 8 items; in 10 items indirect strategies were included. 

Non-literal meaning: In a conversation if a speaker deliberately flouts one of the Grice’s maxims, the speaker 
may be directing the listener to look for a meaning that is different from the meaning that is verbally articulated. 
So, flouting the maxims of quality, quantity, manner and relation results in implicature. Among the 250 tests of 
TOEFL analyzed in this study, nearly half of them assessed non-literal meaning (N=123). Relation maxim was 
flouted more than any other maxims (N=37). Maxims of quality in 22 items and quantity in 23 items were 
flouted. Flouting the maxim of manner was used in 31 items. 

Cultural knowledge: With regard to the use of cultural knowledge in reading section of TOEFL tests, test 
developers did not attempt to include this element of pragmatic knowledge in assessing test takers’ proficiency 
knowledge. 

Q3. To what extent can reading section of IELTS assess pragmatic knowledge? 

The purpose of this research question was to learn about the amount of pragmaticknowledge assessment in 
reading section of IELTS tests. In this study 250 items were analyzed and it was revealed that in 106 items 
pragmatic knowledge has been used (42.4%). It was also shown that 137 times pragmatic knowledge 
components appeared in the items of IELTS tests. In 39 items of IELTS test pragmatic knowledge components 
were used once (15.6%). In some of the items, more than one area of pragmatic knowledge were assessed (N=67, 
26.8%). And in 144 items (57.6%) of items of IELTS, no trace of pragmatic knowledge was found. It can be 
concluded that IELTS is a proficiency test which can assess test takers’ pragmatic knowledge in 42.4% of its 
items. 

Q4. What areas ofpragmatic knowledge are assessed in reading section of IELTS? 

Speech acts: Out of 250 IELTS items analyzed, speech acts have been assess 56 times. Commissives had the 
highest number of speech acts (N=26). Representatives had been used in 21 items and expressives in 6 items. 
And the lowest number of assessed speech acts belongs to directives (N=3). 

Discourse markers: Discourse markers were used in 33 items of IELTS (13.3%). Elaborative discourse markers 
were assessed in 15 items (6%). Contrastive discourse markers and inferential discourse markers each had the 
frequency of 8 (3.2%). The leas discourse markers used is temporal (N=2, 0.8%). 

Politeness strategies: In 11.6% of items in IELTS, politeness functions were included (N=29). Direct strategies 
have been used in 16 items (6.4%) and indirect strategies in 13 items (5.2%). 

Non-literal meaning: In IELTS test, only in 19 items non-literal meaning has been assessed (7.6%). The maxim 
of quantity was assessed in 10 items (4%), and the maxim of manner in 9 items (3.6%). Maxims of quality and 
relevance were not used at all. 
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Q5. To what extent can reading section of TOLIMO assess pragmatic knowledge? 

Finding out the frequency and the percentage of the involvement of pragmatic knowledge in reading section of 
TOLIMO test is the purpose of this research question. In this study 250 items were analyzed and it was revealed 
that in 153 items pragmatic knowledge has been used (61.2%). It was also shown that 362 times pragmatic 
knowledge components appeared in the items of TOLIMO tests. In 59 items of TOLIMO test pragmatic 
knowledge components were used once (23.6%). In some of the items, more than one area of pragmatic 
knowledge were assessed (N=94, 37.6%). And in 97 items (38.8%) of items of TOLIMO, no trace of pragmatic 
knowledge was found. It can be concluded that TOLIMO is a proficiency test which can assess test takers’ 
pragmatic knowledge in 61.2% of its items. 

Q6. What areas ofpragmatic knowledge are assessed in reading section of TOLIMO? 

For TOLIMO test, out of 250 items, 114 instances of speech acts were found. So in a little less than half of the 
items speech acts were used. Declaration was not part of the study, because no case of declaring, cursing, 
sentencing, naming, pronouncing, and blessing was found in these five series of reading section of TOLIMO. 

Discourse markers: Out of the 250 items in TOLIMO test, there were 122 instances of discourse markers. 
Elaborative discourse markers have been used more than other types of discourse markers (N=59). Inferential 
discourse markers with frequency of 19 was the least assessed discourse markers. In 20 items, directive 
discourse markers were assessed. And, temporal discourse markers were used in 24 items in TOLIMO test. 

Politeness strategies: Among 250 items in TOLIMO test, there were 16 items which were related to the 
politeness functions. Direct strategies were used in 9 items; in 7 items indirect strategies were included. 

Non-literal meaning: Among the 250 tests of TOLIMO analyzed in this study, nearly half of them assessed 
non-literal meaning (N=110). Relation maxim was flouted more than any other maxims (N=36). Maxims of 
quality in 20 items and quantity in 23 items were flouted. Flouting the maxim of manner was used in 36 items. 

Q7. Which test assesses pragmatic knowledge of test takers more? IELTS, TOEFL, or TOLIMO? 

TOEFL and IELTS are considered as globally-accepted English proficiency tests, also TOLIMO is a proficiency 
test which is administered locally in IRAN. It is claimed that these three tests assess linguistic knowledge and 
grammatical knowledge, as well as pragmatic knowledge of test takers. But which one assesses test takers’ 
pragmatic knowledge more? 

In this section of the study, components of pragmatic knowledge used in these two tests are compared. 

Speech acts: With regard to overall speech acts, a significant difference was found between the items used in 
TOEFL, IELTS, and TOLIMO. It was found that speech acts are used in 124 items out of 250 items in TOEFL 
test (49.6 %), and 114 times in TOLIMO (45.6%), while 56 out of 250 items of IELTS test assessed speech acts 
(22.4%). So it is evident that TOEFL and TOLIMO are more able to assess test takers’ pragmatic knowledge 
with regard to speech act categories. 

Discourse markers: There was a significant difference between the frequency of IELTS, TOEFL, and TOLIMO 
tests with regard to assessing discourse markers. In TOEFL test 132 out of 250 items assessed discourse markers 
(52.8%), while only 33 out of 208 items of IELTS were involved in assessing test takers’ knowledge of discourse 
markers (13.3%). Also, in 122 items of TOLIMO discourse markers were used (48.2%). As it is clear, for this 
category of pragamatic knowledge, TOEFL and TOLIMO had more instances of discourse markers than IELTS 
did. 

Politeness functions: No significant difference between the frequency of IELTS, TOEFL, and TOLIMO in terms 
of assessing politeness function. The frequency of politeness strategies assessed in TOEFL was 18 (7.2%), and In 
TOLIMO 16 (6.4%), while in 29 items of IELTS politeness strategies have been assessed (11.6%). IELTS is a 
little more able to assess politeness functions of test takers than TOLIMO and TOEFL. 

Non-literal meaning: There is a significant different between the frequency of IELTS, TOEFL, and TOLIMO test 
in terms of assessing non-literal meaning. While Grice’s maxims in TOEFL tests have been flouted 123 times 
(45.2%) and in TOLIMO 110 times (44%), in IELTS test just less than 8 percent of the items assessed 
implicature (N=19, 7.6%). Whereas TOEFL and TOLIMO in which the four maxims have been violated, in 
IELTS two maxims of quantity and manner have been flouted. 

Finally, there is significant difference between the frequency of TOEFL, IELTS, and TOLIMO with regard to 
assessing overall pragmatic knowledge. 518 times pragmatic knowledge has been used in 250 items of TOEFL 
tests, and while the frequency of pragmatic knowledge elements used in IELTS is 130. Also, 362 times different 
areas of pragmatic knowledge have been assessed in TOLIMO. So it is clear TOEFL and TOLIMO are more 
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pragmatically oriented than IELTS. 

7. Conclusion and Implication 

In this study, it was found that there was a significant difference between the frequency of TOEFL, IELTS, and 
TOLIMO with regard to assessing overall pragmatic knowledge. The results showed that to some extent these 
three tests are designed to assess test takers’ pragmatic knowledge, but TOEFL and TOLIMO are more able to 
tap students’ pragmatic knowledge. Also, it was found that all the components of pragmatic knowledge are 
involved in these tests except cultural knowledge. Moreover, among subcategories of speech act, declarations are 
not assessed in these English proficiency tests. 

The present study was conducted to raise the students’ awareness concerning the linguistic and sociolinguistic 
skills in identifying and analyzing the hidden meaning of the writers and to get them familiar with norms of 
pragmatic knowledge in English. The results of the study can be beneficial to many stakeholders. The results of 
this study can be helpful to those who are involved in the process of constructing high stakes test and proficiency 
tests such as IELTS, TOEFL, and TOLIMO. Furthermore this study can be useful for teachers and test 
developers who are making low stake decisions. Also by comparing these three English proficiency tests (IELTS, 
TOEFL, and TOLIMO) according to their ability of each one to assess pragmatic knowledge, proficiency test 
users and administrators can be in a position to choose the best test that can assess communicative competence 
and pragmatic knowledge of test takers. Moreover, the results will show the students and test takers which parts 
of pragmatic knowledge are assessed more in proficiency tests, so that they can focus on those parts. For 
example, if test takers and students were aware that, in proficiency tests, directive speech acts are assessed more, 
they would study these kinds of speech acts more. Or if they knew that cultural knowledge is not part of the 
assessment in universal proficiency tests, they would stop memorizing geographical names or customs in the 
United States. Also, curriculum developers and textbook writers can benefit from this study. 

8. Limitations of the Study 

Like any other study, this thesis is restricted in a number of ways: 

1) The IELTS tests available in the market which were used in this study were for preparation and were not the 
original test administered in IELTS. Although, it is claimed that preparation tests enjoy a high index of reliability 
and validity, and correlation to the original tests. The preparation tests which are used in present study are 
claimed to have been written based on the components of communicative competence which are assessed in 
IELTS. This limitation is also correct for TOLIMO, too. 

2) Since TOEFL test from 2005 up to present were not published and available in the market, the researcher was 
forced to use TOEFL test from 2000 to 2004. 

References 

Austin, J. L. (1962). How to do things with words. Oxford: Oxford University Press. 

Bachman, L. F. (1990). Fundamental Considerations in Language Testing. Oxford: Oxford University Press. 
Bloch, M. (1991). Language, anthropology and cognitive science. Man, 26, 183-197. 

Blum-Kulka, S. (1987). Indirectness and politeness in requests: Same or different? Journal of Pragmatics, 11, 
145-160. 

Brown, P., & Levinson, S. (1987). Politeness: Some universals in language usage. Cambridge, UK: Cambridge 
University Press. 

Corsetti, C. R. (2010). Pragmatic competencein the listening paper of the certificate of proficiency in English. 
BELT Journal, 7(1), 14-25. 

Crystal, D. (2008). A dictionary’ of linguistic and phonetics. Oxford: Blackwell publishing. 

David, O. F. (2008). Socio-Psychological Determinants of ESL Pragmatic Competence in Nigeria. The Social 
Sciences, 3(2), 73-77. 

Donald, M. (1991). The origins of the modern mind: Three stages in the evolution of culture and cognition. 
Cambridge MA: Harvard University Press. 

Educational Testing Service. (2005). TOEFL iBT Tips. Retrieved February 8, 2006, from 

http://upload.mcgill.ca/applying/TOEFL_Tips.pdf 

Flowerdew, J., & Miller, L. (2005). Second language listening: Theory’ and practice. Cambridge: Cambridge 
University Press. 


220 




www.ccsenet.org/elt 


English Language Teaching 


Vol. 8, No. 5; 2015 


Fraser, B. (1999). What are discourse markers? Journal of Pragmatics, 31, 931-952. 

Garcia, P. (2004). Pragmatic comprehension of high and low level language learners. TESL-EJ, 8(2), 1-15. 

Goodenough, W. H. (1957). Cultural Anthropology and Linguistics. In P. L. Garvin (Ed.), Report of the Seventh 
Round Table Meeting on Linguistics and Language Study. Washington, DC: Georgetown University Press. 

Grice, H. P. (1975). Logic and conversation. In P. Cole, & J. Morgan (Eds.), Syntax and Semantics (Vol. 3, pp. 
41-58). New York: Academic Press. 

Holmes, J. (2008). An introduction to sociolinguistics. Harlow: Pearson Education limited. 

IELTS. (2004). IELTS Explained. Retrieved November 28, 2011, from http://www.ielts.org/mediacentre/ 
ieltsexplained/default.aspx 

Jakeman, V., & McDowell, C. (1997). Cambridge practice tests for IELTS 1. Cambridge: Cambridge University 
Press. 

Jakeman, V., & McDowell, C. (2002). Insight into IELTS. Cambridge: Cambridge University Press. 

Jung, J. Y. (2001). Issues in Acquisitional Pragmatics. Working paper in TESOL & Applied Linguistics, 2(3), 
1-13. 

Kirby, J. R. (2006). Reading Comprehension, What have we learned about reading comprehension? Faculty of 
Education, Queen’s University. 

Lakoff, R. (1973). The logic of politeness: Or minding your p’s and q’s. Chicago Linguistic Society, 9, 292-305. 

Leech, G. (1983). Principles of Pragmatics. London: Longman. 

Olson, D. R. (1994). The world on paper: The conceptual and cognitive implications of writing and reading. 
Cambridge UK: Cambridge University Press. 

Quinn, N., & Holland, D. (1987). Cultural Models in Thought and Language. Cambridge: Cambridge University 
Press. 

Rabinowitz, J. D. (1993). A Descriptive Study of the Offer as a Speech Behavior in American English 
(Unpublished doctoral dissertation). University of Pennsylvania, Michigan. 

Searle, J. R. (1969). Speech acts: An essay in the philosophy of language. Cambridge: Cambridge University 
Press. 

Srinarawat, D. (2005). Indirectness as a politeness strategy of Thai speakers. In R. Lakoff, & S. Ide (Eds.), 
Broadening the horizon of linguistic politeness (pp. 175-193). Amsterdam: John Benjamins. 

Tan, P. (1994). Key concepts in ELT. ELT Journal, 48( 1), 100. 

Uso-Juan, E., & Martinez-Flor, A. (2006a). Approaches to language learning and teaching: Towards acquiring 
communicative competence through the four skills. In E. Uso-Juan, & A. Martinez-Flor (Eds.), Current 
Trends in the Development and Teaching of the Four Language Skills (pp. 3-26). Berlin: Mouton de Gruyter. 

Van Dijk, T. (1977). Text and Context. London : Longman. 

Wardhaugh, R. (2008). An Introduction to Sociolinguistics. Oxford: Blackwell. 

Watts, R. J. (2003). Politeness. Cambridge: Cambridge University Press. 

Yule, G. (2000). Pragmatics. Oxford: Oxford University Press. 

Yule, G., & Brown, G (1989). Discourse Analysis. Cambridge: Cambridge University Press. 


Copyrights 

Copyright for this article is retained by the author(s), with first publication rights granted to the journal. 

This is an open-access article distributed under the terms and conditions of the Creative Commons Attribution 
license (http://creativecommons.Org/licenses/by/3.0/). 


221 




